Course Website

The term no one really defined

• What is data science?

Data science is the discipline of making data useful. Ok…so what is it?

Every company claims to be data driven, but they are different…

Every company claims to be data driven, but they are different…

Excerpt from How Airbnb Democratizes Data Science With Data University:

Three tracks of data science

• Engineering (infrastructure and production): the process of making everything else possible

• Analysis: the process of turning raw information into insights in a fast way

• Modeling/Inference: the process of diving deeper into the data to discover the pattern we don’t easily see

(It is a group work from https://github.com/brohrer/academic_advisory/blob/master/authors.md !)

Engineering

1. Data environment: data storage, Kafka platform, Hadoop and Spark cluster etc.

2. Data management: parsing the logs, web scraping, API queries, and interrogating data streams.

3. Production: integrate model and analysis into the production system

Analysis

1. Domain knowledge

2. Exploratory analysis

3. Story telling

Modeling

1. Supervised learning

2. Unsupervised learning

3. Customized model development

General Process of Modeling/Analytics

Some confusions and more to come

• Data infrastructure engineer: Go,Python, AWS/Google Cloud/Azure, logstash, Kafka, and Hadoop
• Data engineer: spark/scala, python, SQL, AWS/Google Cloud/Azure, Data modeling
• BI engineer: Tableau/looker/Mode etc, data visulization, SQL, Python
• Data analyst: SQL, basic statistics, data visualization
• Data scientist: R/Python, SQL, basic + applied statistics, data visualization, experimental design
• Research scientist: R/Python, advanced statistics + experimental design, ML, research background, publications, conference contributions, algorithms
• Applied scientist: Similar to research scientist, ML algorithm design, often with an expectation of basic software engineering skills
• Machine Learning Engineer: More advanced software engineering skillset, algorithms, machine learning algorithm design, system design
• …

Three tracks of data science

Three tracks of data science

Three tracks of data science

Types of Questions (Modeling/Analytics)

• Comparison:
  • Are males more inclined to buy our products than females?
  • Are there any differences in customer satisfaction in different business districts?
  • Do soybean carrying a particular gene have higher oil content?
• Description:
  • Is the distribution of annual income normal?
  • Are there outliers?
  • What are the means of different customer segments?
• Clustering:
  • Which customers have similar product preference?
  • Which printer performs similar pattern to the broken ones?
  • How many different themes are there in the corpus?

Types of Questions (Modeling/Analytics)

• Classification:
  • Who is more likely to buy our product?
  • Is the borrower going to pay back?
  • Is it a spam?
• Regression:
  • What will be the temperature tomorrow?
  • What is the projected net income for the next season?
  • How much inventory should we have?

Data Science Types v.s Needs

Data Flow

Data Science Roles

Startup v.s. Mature company (Pre IPO/Public)

Startup v.s. Mature company (Pre IPO/Public)